[One Workflow](scale): Lazy-load workflow step I/O #253547
[One Workflow](scale): Lazy-load workflow step I/O #253547rosomri merged 24 commits intoelastic:mainfrom
Conversation
…step I/O data when switching to a different execution
… into break_execution_api
| includeInput = true, | ||
| includeOutput = true, |
There was a problem hiding this comment.
When includeInput and includeOutput are omitted (both are optional), the API includes both by default.
From an API semantics perspective, that feels counterintuitive. Optional flags that default to “included” can be surprising, especially if they affect payload size or sensitive data exposure.
Would it make more sense to default both to false, and only include input/output when explicitly requested? That would make the API more explicit and predictable.
There was a problem hiding this comment.
Agreed - I initially set them to true by default for backward compatibility, but you’re right. I’ll switch them to false and share an update in the channel.
💔 Build Failed
Failed CI StepsTest Failures
Metrics [docs]Module Count
Async chunks
History
|
## Summary Reduces memory pressure and network payload size by lazy-loading workflow step execution I/O data instead of fetching it all upfront. https://github.com/user-attachments/assets/2d77d88d-4017-44bd-8581-082717352921 - **Lazy-load step I/O**: Execution polling (`loadExecutionThunk`) now requests lightweight data (`includeInput=false`, `includeOutput=false`). Full step input/output is fetched on demand — when the user clicks a step's tab or hovers a template expression in the YAML editor. - **Server-side source filtering**: `getWorkflowExecution` accepts `includeInput`/`includeOutput` query params and applies `_source_excludes` on Elasticsearch `mget`/`search` calls, avoiding large payloads that can cause OOM. - **Bidirectional React Query cache**: Step I/O fetched by the execution detail panel (`useStepExecution`) or by the YAML editor hover provider share a single cache via `queryClient.setQueryData`, preventing duplicate HTTP requests regardless of access order. - **Cache cleanup on execution switch**: Cached step data is cleared (`removeQueries`) when navigating to a different execution, preventing memory buildup. - **Template hover priority**: Reordered `provideCustomHover` so template expression hovers (`{{ }}`) take precedence over validation decoration tooltips. - **Pure hover enrichment**: Refactored `ensureStepData` → `fetchStepDataIfNeeded` to return enriched data instead of mutating the shared `executionContext` ref. Removed redundant `fetchedStepIds` tracking that caused a caching bug on repeated hovers. - **Extracted `useLazyStepExecutionFetcher` hook**: Moved inline fetch logic out of the YAML editor component into a dedicated hook for readability and testability. - **Narrowed memo deps**: `tabs` memo in `WorkflowStepExecutionDetails` now depends on `hasInput`/`hasError` booleans instead of the full `stepExecution` object. ### Example flows **1. Execution polling — lightweight, no I/O** ``` GET /api/workflowExecutions/exec-123?includeInput=false&includeOutput=false ``` Returns execution metadata and step statuses/durations, but `input` and `output` fields are excluded at the Elasticsearch `_source` level. This runs every poll cycle. **2. Hovering a template expression — lazy fetch + cache** User hovers `{{ steps.search.output.hits }}` in the YAML editor: ``` 1. Hover provider calls fetchStepExecutionData("search") 2. Hook maps "search" → step doc ID "step-doc-abc" 3. React Query cache miss → GET /api/workflowExecutions/exec-123/steps/step-doc-abc 4. Response stored in cache: queryClient.setQueryData(["stepExecution", "exec-123", "step-doc-abc"], data) 5. Hover tooltip shows the resolved value Second hover on the same step (or any steps.search.* expression): 1. fetchStepExecutionData("search") → cache hit → no HTTP request 2. Hover tooltip shows the resolved value immediately ``` For terminal steps, `useStepExecution` uses `staleTime: Infinity` — the cached data never goes stale for the lifetime of that execution. **3. Opening the I/O tab — served from cache** After the hover above already fetched `step-doc-abc`, user clicks the step and opens the Output tab: ``` 1. useStepExecution("exec-123", "step-doc-abc", "completed") runs 2. React Query finds ["stepExecution", "exec-123", "step-doc-abc"] in cache 3. No HTTP request — data renders immediately ``` This works in both directions: if the user clicks the Output tab first, the hover provider finds the data in cache on subsequent hovers. **4. Switching execution — cache cleanup** ``` 1. User selects execution "exec-456" 2. useEffect cleanup fires: queryClient.removeQueries({ queryKey: ["stepExecution", "exec-123"] }) 3. All cached step I/O for the previous execution is evicted 4. Fresh lightweight polling starts for "exec-456" ``` ## Test plan - [x] `get_workflow_execution.test.ts` — Verifies `_source_excludes` is correctly passed to `esClient.mget` and `searchStepExecutions` based on `includeInput`/`includeOutput` flags - [x] `get_workflow_execution_by_id.test.ts` — Updated existing route tests; added cases verifying query params are parsed and forwarded to the API layer - [x] `use_step_execution.test.ts` — Verifies `staleTime: Infinity` and no polling for terminal steps; polling at 5s for running steps; polling stops on status transition - [x] `workflow_execution_detail.test.tsx` — Verifies `removeQueries` is called on unmount and when `executionId` changes - [x] `unified_hover_provider.test.ts` — Verifies hover values persist across multiple invocations, enrichment skipped when output already present, graceful fallback when fetch returns null - [x] `workflow_yaml_editor.test.tsx` — Updated test wrapper to include `QueryClientProvider` for `useLazyStepExecutionFetcher` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
…load I/O change (#254087) ## Summary Fixes workflow execution output retrieval for agent-builder consumers. After [#253547](#253547) changed `getWorkflowExecution` to exclude step I/O by default, `getExecutionState` was no longer receiving step outputs - causing `getWorkflowOutput` to always return `null`. This passes `includeOutput: true` explicitly so the output is available when the execution completes.
Summary
Reduces memory pressure and network payload size by lazy-loading workflow step execution I/O data instead of fetching it all upfront.
Screen.Recording.2026-02-18.at.13.18.51.mov
loadExecutionThunk) now requests lightweight data (includeInput=false,includeOutput=false). Full step input/output is fetched on demand — when the user clicks a step's tab or hovers a template expression in the YAML editor.getWorkflowExecutionacceptsincludeInput/includeOutputquery params and applies_source_excludeson Elasticsearchmget/searchcalls, avoiding large payloads that can cause OOM.useStepExecution) or by the YAML editor hover provider share a single cache viaqueryClient.setQueryData, preventing duplicate HTTP requests regardless of access order.removeQueries) when navigating to a different execution, preventing memory buildup.provideCustomHoverso template expression hovers ({{ }}) take precedence over validation decoration tooltips.ensureStepData→fetchStepDataIfNeededto return enriched data instead of mutating the sharedexecutionContextref. Removed redundantfetchedStepIdstracking that caused a caching bug on repeated hovers.useLazyStepExecutionFetcherhook: Moved inline fetch logic out of the YAML editor component into a dedicated hook for readability and testability.tabsmemo inWorkflowStepExecutionDetailsnow depends onhasInput/hasErrorbooleans instead of the fullstepExecutionobject.Example flows
1. Execution polling — lightweight, no I/O
Returns execution metadata and step statuses/durations, but
inputandoutputfields are excluded at the Elasticsearch_sourcelevel. This runs every poll cycle.2. Hovering a template expression — lazy fetch + cache
User hovers
{{ steps.search.output.hits }}in the YAML editor:For terminal steps,
useStepExecutionusesstaleTime: Infinity— the cached data never goes stale for the lifetime of that execution.3. Opening the I/O tab — served from cache
After the hover above already fetched
step-doc-abc, user clicks the step and opens the Output tab:This works in both directions: if the user clicks the Output tab first, the hover provider finds the data in cache on subsequent hovers.
4. Switching execution — cache cleanup
Test plan
get_workflow_execution.test.ts— Verifies_source_excludesis correctly passed toesClient.mgetandsearchStepExecutionsbased onincludeInput/includeOutputflagsget_workflow_execution_by_id.test.ts— Updated existing route tests; added cases verifying query params are parsed and forwarded to the API layeruse_step_execution.test.ts— VerifiesstaleTime: Infinityand no polling for terminal steps; polling at 5s for running steps; polling stops on status transitionworkflow_execution_detail.test.tsx— VerifiesremoveQueriesis called on unmount and whenexecutionIdchangesunified_hover_provider.test.ts— Verifies hover values persist across multiple invocations, enrichment skipped when output already present, graceful fallback when fetch returns nullworkflow_yaml_editor.test.tsx— Updated test wrapper to includeQueryClientProviderforuseLazyStepExecutionFetcher